DATA ENGINEERING / BATCH + CDC PIPELINE
VeloCiSync M2MR is a dual-path pipeline where Spark replays historical GitHub-event backfills through MySQL staging while Debezium and Kafka stream canonical database changes into MongoDB and Redis — engineered so that re-running any step never corrupts the result.
01 / CONTEXT & PROBLEM
Historical backfills get re-run — after a crash, a bug fix, a partial load — and naive re-runs create duplicates. Live CDC has the opposite failure: if the consumer commits its Kafka offset before a downstream write lands, a crash silently loses events. Most tutorial pipelines handle neither.
The goal: one system where MySQL is the single canonical store, historical JSONL replays are provably idempotent, and CDC projections into MongoDB and Redis are exactly-once in effect — with the audit trail to prove it.
02 / ARCHITECTURE
PATH A · HISTORICAL BACKFILL (solid)
GitHub events JSONL
438,018 events · 24 hourly partitions
→
Spark processing
contract validation · rejected-record quarantine
→
MySQL staging
SHA-256 fingerprints · idempotent merges · dedup
→
MySQL canonical
immutable event IDs · audit accounting
PATH B · NEAR-REAL-TIME CDC (dashed)
MySQL binlog
canonical changes
→
Debezium → Kafka
change topics
→
Checkpoint-safe consumer
offset commits ONLY after both targets succeed
→
MongoDB + Redis
document & low-latency projections
Entire stack containerized with Docker Compose · Makefile automation · CI validation · operational runbooks · 14 unit/integration tests
03 / KEY DECISIONS
Idempotency by construction, not by cleanup
Immutable event IDs + SHA-256 payload fingerprints + staged merges mean any partition can be replayed at any time — duplicates are detected and conflicts protected at write time, never repaired afterwards.
Offsets advance only after BOTH projections land
The CDC consumer treats MongoDB and Redis as one atomic outcome: a crash between writes replays the record instead of losing it. At-least-once delivery + idempotent writes = exactly-once effect.
MySQL as the single canonical store
MongoDB and Redis are disposable projections, rebuildable from MySQL at any time. That one rule collapses the hardest distributed-systems questions into "replay the log".
Audit accounting as a first-class output
Every batch reports accepted / rejected / duplicate counts that must reconcile to input totals — the same discipline as financial bookkeeping, applied to events.
04 / RESULTS
An analytics notebook on top covers activity volume, actor/repository concentration, hourly patterns, and KPI quality gates with dashboard-ready outputs.
05 / WHAT I'D DO DIFFERENTLY
The 46.33 rec/s figure comes from a single-consumer local batch test. The next iteration gets consumer-group parallelism with per-partition ordering guarantees, plus load tests that measure how the checkpoint-safe pattern behaves when one target (Redis vs MongoDB) is degraded.
I'd also promote the audit accounting into queryable tables with alerting, so reconciliation failures page someone instead of living in logs.